Outcomes and Costs of the Transition From a Paper-Based Immunization System to a Digital Immunization System in Vietnam: Mixed Methods Study.

BACKGROUND: The electronic National Immunization Information System (NIIS) was introduced nationwide in Vietnam in 2017. Health workers were expected to use the NIIS alongside the legacy paper-based system. Starting in 2018, Hanoi and Son La provinces transitioned to paperless reporting. Interventions to support this transition included data guidelines and training, internet-based data review meetings, and additional supportive supervision visits. OBJECTIVE: This study aims to assess (1) changes in NIIS data quality and use, (2) changes in immunization program outcomes, and (3) the economic costs of using the NIIS versus the traditional paper system. METHODS: This mixed methods study took place in Hanoi and Son La provinces. It aimed to analyses pre- and postintervention data from various sources including the NIIS; household and health facility surveys; and interviews to measure NIIS data quality, data use, and immunization program outcomes. Financial data were collected at the national, provincial, district, and health facility levels through record review and interviews. An activity-based costing approach was conducted from a health system perspective. RESULTS: NIIS data timeliness significantly improved from pre- to postintervention in both provinces. For example, the mean number of days from birth date to NIIS registration before and after intervention dropped from 18.6 (SD 65.5) to 5.7 (SD 31.4) days in Hanoi (P<.001) and from 36.1 (SD 94.2) to 11.7 (40.1) days in Son La (P<.001). Data from Son La showed that the completeness and accuracy improved, while Hanoi exhibited mixed results, possibly influenced by the COVID-19 pandemic. Data use improved; at postintervention, 100% (667/667) of facilities in both provinces used NIIS data for activities beyond monthly reporting compared with 34.8% (202/580) in Hanoi and 29.4% (55/187) in Son La at preintervention. Across nearly all antigens, the percentage of children who received the vaccine on time was higher in the postintervention cohort compared with the preintervention cohort. Up-front costs associated with developing and deploying the NIIS were estimated at US $0.48 per child in the study provinces. The commune health center level showed cost savings from changing from the paper system to the NIIS, mainly driven by human resource time savings. At the administrative level, incremental costs resulted from changing from the paper system to the NIIS, as some costs increased, such as labor costs for supportive supervision and additional capital costs for equipment associated with the NIIS. CONCLUSIONS: The Hanoi and Son La provinces successfully transitioned to paperless reporting while maintaining or improving NIIS data quality and data use. However, improvements in data quality were not associated with improvements in the immunization program outcomes in both provinces. The COVID-19 pandemic likely had a negative influence on immunization program outcomes, particularly in Hanoi. These improvements entail up-front financial costs.

How digital transformation can accelerate data use in health systems.

Introduction: We are in an era of rapid technological advance and digitalization. Countries around the world want to leverage technology to improve health outcomes by accelerating data use and increasing evidence-based decision-making to inform action in the health sector. Yet, there is no "one size fits all" approach to achieving this. To understand more, PATH and Cooper/Smith conducted a study documenting and analyzing the experiences of five African countries (Burkina Faso, Ethiopia, Malawi, South Africa, and Tanzania) that are on this digitalization journey. The goal was to examine their different approaches and develop a holistic model of digital transformation for data use that identifies what the essential components for digitalization success are and how they interact with each other. Methods: Our research had two phases: first, we analyzed documentation from the five countries to identify core components and enabling factors for successful digital transformation, as well as barriers encountered; and second, we held interviews with key informants and focus groups within the countries to fill gaps and validate findings. Findings: Our findings show that the core components of digital transformation success are highly interrelated. We found that the more successful digitalization efforts address issues that cut across components-such as stakeholder engagement, health workforce capacity, and governance structures-and consider more than just systems and tools. Specifically, we found two critical components of digital transformation that have not been addressed in previous models like the eHealth strategy building blocks developed by the World Health Organization and the International Telecommunication Union: (a) cultivating a culture of data use throughout the health sector and (b) managing the process of system-wide behavior change required to move from manual or paper-based to digital systems. Conclusion: The resulting model is based on the study's findings and is intended to inform low- and middle-income (LMIC) country governments, global policymakers (such as WHO), implementers, and funders. It provides specific, concrete, evidence-based strategies these key stakeholders can implement to improve digital transformation for data use in health systems, planning, and service delivery.

Design, Development, and Deployment of an Electronic Immunization Registry: Experiences From Vietnam, Tanzania, and Zambia.

INTRODUCTION: There is growing interest among low- and middle-income countries to introduce electronic immunization registries (EIRs) that capture individual-level vaccine data. We compare the design, development, and deployment of EIRs in Vietnam, Tanzania, and Zambia. Through desk review and the authors' firsthand implementation experiences, we describe experiences related to timeline, partnerships, financial costs, and technology and infrastructure. IMPLEMENTATION EXPERIENCE: The country cases highlight the multi-year timeline required to implement an EIR at scale and the benefit of multiple iterative cycles to pilot and redesign the system before achieving scale. Of the 3 countries, only Vietnam has achieved nationwide scale of the EIR, which took 7 years. In all 3 countries, national government leadership as part of an interdisciplinary team (with experience in leadership, technology, and immunization) was important to ensure country ownership and sustainability. Where international software developers were contracted, partnering with a local software company helped improve responsiveness and sustainability. Across all 3 countries, governments contributed significant in-kind time in addition to investments from donors. Cost savings were observed in Tanzania and Zambia, largely driven by health worker time savings from using the EIR. All 3 case countries underscore the need to understand the local technology and infrastructure context and design the EIR to fit the context. In Vietnam, an initial landscape assessment was conducted to assess technology and infrastructure, whereas in Tanzania and Zambia, user advisory groups provided insights. Existing infrastructure informed EIR design decisions, such as choosing a system with offline functionality in Tanzania and Zambia. All 3 countries have a local partner to provide ongoing technical support. CONCLUSION: Comparing implementation factors across these cases highlights practical experience and recommendations that complement existing EIR guidance documents. The findings and recommendations from this study can inform other countries considering or in the process of implementing an EIR.

Changes in on-time vaccination following the introduction of an electronic immunization registry, Tanzania 2016-2018: interrupted time-series analysis.

BACKGROUND: Digital health interventions (DHI) have the potential to improve the management and utilization of health information to optimize health care worker performance and provision of care. Despite the proliferation of DHI projects in low-and middle-income countries, few have been evaluated in an effort to understand their impact on health systems and health-related outcomes. Although more evidence is needed on their impact and effectiveness, the use of DHIs among immunization programs has become more widespread and shows promise for improving vaccination uptake and adherence to immunization schedules. METHODS: Our aim was to assess the impact of an electronic immunization registry (EIR) using an interrupted time-series analysis to analyze the effect on proportion of on-time vaccinations following introduction of an EIR in Tanzania. We hypothesized that the introduction of the EIR would lead to statistically significant changes in vaccination timeliness at 3, 6, and > 6 months post-introduction. RESULTS: For our primary analysis, we observed a decrease in the proportion of on-time vaccinations following EIR introduction. In contrast, our sensitivity analysis estimated improvements in timeliness among those children with complete vaccination records. However, we must emphasize caution interpreting these findings as they are likely affected by implementation challenges. CONCLUSIONS: This study highlights the complexities of using digitized individual-level routine health information system data for evaluation and research purposes. EIRs have the potential to improve vaccination timeliness, but analyses using EIR data can be complicated by data quality issues and inconsistent data entry leading to difficulties interpreting findings.

Added Value of Electronic Immunization Registries in Low- and Middle-Income Countries: Observational Case Study in Tanzania.

BACKGROUND: There is growing interest and investment in electronic immunization registries (EIRs) in low- and middle-income countries. EIRs provide ready access to patient- and aggregate-level service delivery data that can be used to improve patient care, identify spatiotemporal trends in vaccination coverage and dropout, inform resource allocation and program operations, and target quality improvement measures. The Government of Tanzania introduced the Tanzania Immunization Registry (TImR) in 2017, and the system has since been rolled out in 3736 facilities in 15 regions. OBJECTIVE: The aims of this study are to conceptualize the additional ways in which EIRs can add value to immunization programs (beyond measuring vaccine coverage) and assess the potential value-add using EIR data from Tanzania as a case study. METHODS: This study comprised 2 sequential phases. First, a comprehensive list of ways EIRs can potentially add value to immunization programs was developed through stakeholder interviews. Second, the added value was evaluated using descriptive and regression analyses of TImR data for a prioritized subset of program needs. RESULTS: The analysis areas prioritized through stakeholder interviews were population movement, missed opportunities for vaccination (MOVs), continuum of care, and continuous quality improvement. The included TImR data comprised 958,870 visits for 559,542 patients from 2359 health facilities. Our analyses revealed that few patients sought care outside their assigned facility (44,733/810,568, 5.52% of applicable visits); however, this varied by region; facility urbanicity, type, ownership, patient volume, and duration of TImR system use; density of facilities in the immediate area; and patient age. Analyses further showed that MOVs were highest among children aged <12 months (215,576/831,018, 25.94% of visits included an MOV and were applicable visits); however, there were few significant differences based on other individual or facility characteristics. Nearly half (133,337/294,464, 45.28%) of the children aged 12 to 35 months were fully vaccinated or had received all doses except measles-containing vaccine-1 of the 14-dose under-12-month schedule (ie, through measles-containing vaccine-1), and facility and patient characteristics associated with dropout varied by vaccine. The continuous quality improvement analysis showed that most quality issues (eg, MOVs) were concentrated in <10% of facilities, indicating the potential for EIRs to target quality improvement efforts. CONCLUSIONS: EIRs have the potential to add value to immunization stakeholders at all levels of the health system. Individual-level electronic data can enable new analyses to understand service delivery or care-seeking patterns, potential risk factors for underimmunization, and where challenges occur. However, to achieve this potential, country programs need to leverage and strengthen the capacity to collect, analyze, interpret, and act on the data. As EIRs are introduced and scaled in low- and middle-income countries, implementers and researchers should continue to share real-world examples and build an evidence base for how EIRs can add value to immunization programs, particularly for innovative uses.

Determinants of Facility-Level Use of Electronic Immunization Registries in Tanzania and Zambia: An Observational Analysis.

BACKGROUND: As more countries transition from paper-based to electronic immunization registries (EIRs) to collect and track individual immunization data, guidance is needed for successful adoption and use of these systems. Little research is available on the determinants of EIR use soon after introduction. This observational study assesses the determinants of facility health care workers' use of new EIRs in Tanzania and Zambia, implemented during 2016 to 2018. METHODS: We used EIR data entered between 2016 and 2018 from 3 regions in Tanzania and 1 province in Zambia to measure weekly EIR system use for a total of 50,639 facility-weeks. We joined secondary data on facility characteristics and applied the Performance of Routine Information System Management framework to categorize characteristics as organizational, technical, or behavioral. We used a generalized estimating equations logistic regression model to assess facility characteristics as potential determinants of system use. RESULTS: In both countries, the estimated odds of weekly EIR use declined weekly after EIR introduction. In Tanzania, health centers and hospitals had increased odds of system use compared to dispensaries. For each additional health care worker trained in a facility during the EIR introduction, the estimated odds of weekly EIR use increased. Tanzanian facilities that had transitioned entirely to paperless reporting had higher odds of sustained use compared to those maintaining parallel electronic and paper-based reporting systems. In Zambia, distance from the district health office was significantly associated with decreasing odds of system use. There were significant differences in EIR use by district in both countries. DISCUSSION: The results highlight the importance of organizational and behavioral factors in explaining sustained EIR use. As EIRs are introduced in new settings, we recommend indicators of engagement and use be built directly into the system so they can be routinely monitored, and course corrections can be implemented as needed.

Determinants of Scale-up From a Small Pilot to a National Electronic Immunization Registry in Vietnam: Qualitative Evaluation.

BACKGROUND: Digital health innovations can improve health system performance, yet previous experience has shown that many innovations do not advance beyond the pilot stage to achieve scale. Vietnam's National Immunization Information System (NIIS) began as a series of digital health pilots, first initiated in 2010, and was officially launched nationwide in 2017. The NIIS is one of the few examples of an electronic immunization registry (EIR) at national scale in low- and middle-income countries. OBJECTIVE: The aim of this study was to understand the determinants of scale-up of the national EIR in Vietnam. METHODS: This qualitative study explored the facilitators and barriers to national scale-up of the EIR in Vietnam. Qualitative data were collected from October to December 2019 through in-depth key informant interviews and desk review. The mHealth Assessment and Planning for Scale (MAPS) Toolkit guided the development of the study design, interview guides, and analytic framework. MAPS defines the key determinants of success, or the "axes of scale," to be groundwork, partnerships, financial health, technology and architecture, operations, and monitoring and evaluation. RESULTS: The partnership and operations axes were critical to the successful scale-up of the EIR in Vietnam, while the groundwork and monitoring and the evaluation axes were considered to be strong contributors in the success of all the other axes. The partnership model leveraged complementary strengths of the technical working group partners: the Ministry of Health General Department of Preventive Medicine, the National Expanded Program on Immunization, Viettel (the mobile network operator), and PATH. The operational approach to introducing the NIIS with lean, iterative, and integrated training and supervision was also a key facilitator to successful scale-up. The financial health, technology and architecture, and operations axes were identified as barriers to successful deployment and scale-up. Key barriers to scale-up included insufficient estimates of operational costs, unanticipated volume of data storage and transmission, lack of a national ID to support interoperability, and operational challenges among end users. Overall, the multiple phases of EIR deployment and scale-up from 2010 to 2017 allowed for continuous learning and improvement that strengthened all the axes and contributed to successful scale-up. CONCLUSIONS: The results highlight the importance of the measured, iterative approach that was taken to gradually expand a series of small pilots to nationwide scale. The findings from this study can be used to inform other countries considering, introducing, or in the process of scaling an EIR or other digital health innovations.

Perceptions of factors influencing the introduction and adoption of electronic immunization registries in Tanzania and Zambia: a mixed methods study.

BACKGROUND: As technology has become cheaper and more accessible, health programs are adopting digital health interventions (DHI) to improve the provision of and demand for health services. These interventions are complex and require strong coordination and support across different health system levels and government departments, and they need significant capacities in technology and information to be properly implemented. Electronic immunization registries (EIRs) are types of DHI used to capture, store, access, and share individual-level, longitudinal health information in digitized records. The BID Initiative worked in partnership with the governments of Tanzania and Zambia to introduce an EIR at the sub-national level in both countries within 5 years as part of a multi-component complex intervention package focusing on data use capacity-building. METHODS: We aimed to gather and describe learnings from the BID experience by conducting a framework-based mixed methods study to describe perceptions of factors that influenced scale-up of the EIR. Data were collected through key informant interviews, a desk review, EIRs, and health management information systems. We described how implementation of the EIRs fulfilled domains described in our conceptual framework and used cases to illustrate the relationships and relative influence of domains for scale-up and adoption of the EIR. RESULTS: We found that there was no single factor that seemed to influence the introduction or sustained adoption of the EIR as many of the factors were interrelated. For EIR introduction, strong strategic engagement among partners was important, while EIR adoption was influenced by adequate staffing at facilities, training, use of data for supervision, internet and electricity connectivity, and community sensitization. CONCLUSIONS: Organizations deploying DHIs in the future should consider how best to adapt their intervention to the existing ecosystem, including human resources and organizational capacity, as well as the changing technological landscape during planning and implementation.

The impact of an integrated electronic immunization registry and logistics management information system (EIR-eLMIS) on vaccine availability in three regions in Tanzania: A pre-post and time-series analysis.

BACKGROUND: Since 2016, the Government of Tanzania has been implementing TImR, an integrated Electronic Immunization registry-logistics management information system (EIR-LMIS) that includes stock notifications. The objective of this study is to estimate the impact of this intervention on vaccine availability. METHODS: Monthly stock-out data were collected from paper registers at facilities, an Excel-based system at districts, and the new system (TImR) across all 924 health facilities in Arusha, Tanga and Kilimanjaro Regions. Six months of stockout rates pre- and post-introduction, by antigen, were compared via a two-way analysis of variance (ANOVA). A mixed-effects logistic regression model with the TImR data identified predictors of vaccine availability across antigens. FINDINGS: Post-introduction, ANOVA models estimated that overall stock-out rates declined from a monthly average of 7.1% to 2.1% (p < 0.01). Three specific vaccines had fewer stock-outs; OPV's monthly average dropped from 12.5% to 2.1% (p < 0.01), MR from 9.4% to 1.0% (p < 0.01) and DTP-HepB-HiB from 8.1% to 1.7% (p < 0.01). In the mixed-effects logistic regression model, controlling for antigen, odds of stock-out were 4.1% (95% CI: 3.3 - 4.9) lower for each week of tenure. Compared to DTP-HepB-HiB vaccine, odds of BCG vaccine being stocked out were 4.31 as high (95% CI: 3.1 - 5.0). The odds of being stocked-out were 29.7% lower for PCV (95% CI: 8.8 - 45.8) and 26.6% (95% CI: 3.4 - 44.1) lower for rotavirus vaccines compared to DTP-HepB-HiB. The odds of stock out were 37.7% lower for MR vaccine than DTP-HepB-HiB (95% CI: 18.1 - 52.6). CONCLUSIONS: Tanzania's integrated EIR-eLMIS may increase vaccine availability compared to its paper and Excel based system. Post-introduction of an eLMIS, the odds of a vaccine stock-out reduced over time. Further research could determine the impact of this intervention on vaccine wastage and replenishment response times.

The costs of developing, deploying and maintaining electronic immunisation registries in Tanzania and Zambia.

OBJECTIVE: To determine the costs to develop, roll out and maintain electronic immunisation registries (EIRs) and a related suite of data use interventions. METHODS: The Better Immunisation Data (BID) Initiative conducted the activities from 2013 to 2018 in three regions in Tanzania and one province in Zambia. The Initiative's financial records were used to account for the financial costs of designing and developing the EIRs, BID staff time, expenditures for rolling out the EIR systems and the related suite of interventions to health facilities, and recurrent costs. Total financial costs, cost per facility and cost per child were calculated in 2018 US$. FINDINGS: Total expenditures were ~US$4.2 million in Tanzania and US$3.6 million in Zambia. System design and development costs accounted for ~33% and 26% of the expenditures in each country, respectively, while BID staff costs accounted for 39% and 52%, respectively. Average expenditures per health facility for rolling out the EIR system were between US$709 and US$1320 for the Tanzania regions and US$2591 for Zambia. The annualised average expenditure per child was estimated to be between US$3.30 and US$3.81 for the regions in Tanzania and US$8.46 in Zambia. Expenditures per child were higher in Zambia partly because of a much smaller birth cohort compared with Tanzania. CONCLUSION: Other countries may benefit from the investments made and lessons learnt in Tanzania and Zambia by leveraging these now existing EIR platforms and rollout strategies, and hence may be able to implement EIRs at lower costs than reported here.

Three Waves of Data Use Among Health Workers: The Experience of the Better Immunization Data Initiative in Tanzania and Zambia.

The governments of Tanzania and Zambia identified key data-related challenges affecting immunization service delivery including identifying children due for vaccines, time-consuming data entry processes, and inadequate resources. To address these challenges, since 2014, the countries have partnered with PATH's Better Immunization Data Initiative to design and deploy a suite of data quality and use interventions. Two key aspects of the interventions were an electronic immunization registry and tools and practices to strengthen a culture of data use. As both countries deployed the interventions, 3 distinct changes in data use emerged organically. This article provides a detailed summary of these 3 phases or waves, based mostly on qualitative data or observation: (1) strengthening data collection using new data collection tools and processes and increasing efficiency of health workers; (2) improving data quality regarding accuracy and completeness; and (3) increasing use of data to take action to strengthen their work and for programmatic decision making. These waves clearly demonstrated the growing ability of health workers to move from data collectors to data analyzers who began to focus on the data quality and then the value of using the data in their day-to-day activities.

Electronic Immunization Registries in Tanzania and Zambia: Shaping a Minimum Viable Product for Scaled Solutions.

As part of the work the Better Immunization Data (BID) Initiative undertook starting in 2013 to improve countries' collection, quality, and use of immunization data, PATH partnered with countries to identify the critical requirements for an electronic immunization registry (EIR). An EIR became the core intervention to address the data challenges that countries faced but also presented complexities during the development process to ensure that it met the core needs of the users. The work began with collecting common system requirements from 10 sub-Saharan African countries; these requirements represented the countries' vision of an ideal system to track individual child vaccination schedules and elements of supply chain. Through iterative development processes in both Tanzania and Zambia, the common requirements were modified and adapted to better fit the country contexts and users' needs, as well as to be developed with the technology available at the time. This process happened across four different software platforms. This paper outlines the process undertaken and analyzes similarities and differences across the iterations of the EIR in both countries, culminating in the development of a registry in Zambia that includes the most critical aspects required for initially deploying the registry and embodies what could be considered the minimum viable product for an EIR.

Homozygous loss-of-function variants of TASP1, a gene encoding an activator of the histone methyltransferases KMT2A and KMT2D, cause a syndrome of developmental delay, happy demeanor, distinctive facial features, and congenital anomalies.

We report four unrelated children with homozygous loss-of-function variants in TASP1 and an overlapping phenotype comprising developmental delay with hypotonia and microcephaly, feeding difficulties with failure-to-thrive, recurrent respiratory infections, cardiovascular malformations, cryptorchidism, happy demeanor, and distinctive facial features. Two children had a homozygous founder deletion encompassing exons 5-11 of TASP1, the third had a homozygous missense variant, c.701 C>T (p.Thr234Met), affecting the active site of the encoded enzyme, and the fourth had a homozygous nonsense variant, c.199 C>T (p.Arg67*). TASP1 encodes taspase 1 (TASP1), which is responsible for cleaving, thus activating, the lysine methyltransferases KMT2A and KMT2D, which are essential for histone methylation and transcription regulation. The consistency of the phenotype, the critical biological function of TASP1, the deleterious nature of the TASP1 variants, and the overlapping features with Wiedemann-Steiner and Kabuki syndromes respectively caused by pathogenic variants in KMT2A and KMT2D all support that TASP1 is a disease-related gene.

Redefining vaccination coverage and timeliness measures using electronic immunization registry data in low- and middle-income countries.

Vaccine coverage is routinely used as a performance indicator for immunization programs both at local and global levels. For many national immunization programs, there are challenges with accurately estimating vaccination coverage based on available data sources, however an increasing number of low- and middle-income countries (LMICs) have begun implementing electronic immunization registries to replace health facilities' paper-based tools and aggregate reporting systems. These systems allow for more efficient capture and use of routinely reported individual-level data that can be used to calculate dose-specific and cohort vaccination coverage, replacing the commonly used aggregate routine health information system data. With these individual-level data immunization programs have the opportunity to redefine performance measures to enhance programmatic decision-making at all levels of the health system. In this commentary, we discuss how measures for assessing vaccination status and program performance can be redefined and recalculated using these data when generated at the health facility level and the implications of the use and availability of electronic individual-level data.

Accelerating Harmonization in Digital Health.

Digital tools play an important role in supporting front-line health workers who deliver primary care. This paper explores the current state of efforts undertaken to move away from single-purpose applications of digital health towards integrated systems and solutions that align with national strategies. Through examples from health information systems, data and health worker training, this paper demonstrates how governments and stakeholders are working to integrate digital health services. We emphasize three factors as crucial for this integration: development and implementation of national digital health strategies; technical interoperability and collaborative approaches to ensure that digital health has an impact on the primary care level. Consolidation of technologies will enable an integrated, scaleable approach to the use of digital health to support health workers. PURPOSE: As this edition explores a paradigm shift towards harmonization in primary healthcare systems, this paper explores complementary efforts undertaken to move away from single-purpose applications of digital health towards integrated systems and solutions that align with national strategies. It describes a paradigm shift towards integrated and interoperable systems that respond to health workers' needs in training, data and health information; and calls for the consolidation and integration of digital health tools and approaches across health areas, functions and levels of the health system. It then considers the critical factors that must be in place to support this paradigm shift. This paper aims not only to describe steps taken to move from fractured pilots to effective systems, but to propose a new perspective focused on consolidation and collaboration guided by national digital health strategies.